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Summary of Review 



Choice and Education across the States, published by the Heartland Institute, is an ad- 
vocacy document that assigns letter grades to states based on the extensiveness of each 
state’s school choice system. The report asserts, based on a faulty use of past research, that 
an increase in school choice will strengthen accountability and improve student achieve- 
ment. It awards most states low grades, reflecting a desire for more school choice through- 
out the nation. But the report does not provide much in the way of useful information; it 
only offers the argument that states should increase school choice, dressed up with a letter 
grade for each state. 
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Review 



I. Introduction 

Choice and Education across the States 1 
presents a report card grading and ranking 
the states in terms of school choice efforts. 
The report contends that states should pro- 
vide parents with more school choice op- 
tions as a way to increase accountability and 
improve achievement. It examines four as- 
pects of school choice: vouchers, charter 
schools, tax credits, and public school 
choice. 

This report card was produced by the Heart- 
land Institute, whose mission is to “promote 
free-market solutions” that include “parental 

9 

choice in education.” Consistent with that 
mission, the new report advocates an expan- 
sion of the school choice marketplace, as 
well as fewer constraints on all forms of 
school choice. Some aspects of the report 
itself are straightforward in that advocacy, 
but — as discussed below — other aspects are 
misleading. 

II. Findings And Conclusions 
Of The Report 

The report card gives higher grades to states 
that have adopted more school choice pro- 
grams, particularly programs with a stronger 
free-market orientation. It claims that school 
choice has been effective, so states willing 
to try these programs can improve the qual- 
ity of their schools. 

The particular findings and conclusions of 
the report can be thought of as the individual 
state grades. But more generally the two 
main conclusions reached are as follows: 

• States with more types of school 
choice received high grades, while 
those that do not emphasize choice 



were ranked at the bottom. The states 
receiving the most points are Arizona 
(15 points), Florida (14 points), Wiscon- 
sin (12 points), and Ohio (11 points) — 
states that offer a variety of different 
choice programs. The eight lowest- 
scoring states (West Virginia, South Da- 
kota, Oklahoma, North Dakota, Ne- 
braska, Alabama, Washington, and Mon- 
tana) each were awarded only one or two 
points since they only offer public 
school choice: magnet schools, virtual 
schools, or open enrollment. 

• Although a wide range of school 
choice policies and approaches can be 
found across the 50 states, the report 
places most states at the bottom of the 
grade distribution. There are almost 
three times as many states at the bottom 
than at the top of the grade distribution. 
No state received a grade of an “A” on 
the report’s non-curved scale. That is, 
even Arizona is seen by the Heartland 
report as needing to expand its choice 
offerings. 

III. Rationales supporting Findings 
and Conclusions of the Report 

As described below, the report clearly sets 
forth the criteria selected for awarding 
grades, but the findings and conclusions 
were not, in any meaningful way, “ana- 
lyzed.” To support the first conclusion that 
the highest scores should be given to states 
that have a wider variety of school choice 
programs, the report puts forth the argument 
that once school choice is introduced into 
state policy, its popularity fosters expansion 
and its opponents find it more difficult to 
foil choice plans. 

To support the second conclusion, that most 
states deserve low grades because they have 
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insufficient school choice programs, the 
report points to “highly entrenched special 
interests, both union and corporate, who 
profit from the status quo” (p. 11). 

Accordingly, these conclusions are 
grounded in strongly felt rhetorical argu- 
ments, but a foe of market-based school 
policies might use the same criteria to re- 
verse the grades (e.g., Washington and Mon- 
tana given As, and Arizona and Florida 
given Fs). The values and beliefs of the 
Heartland author are, effectively, the com- 
plete rationale supporting these findings and 
conclusions. 

IV. The Report’s Use of 
Research Literature 

The empirical contentions in the report 
mainly concern the past effects of school 
choice policies. But it should be noted that 
this report includes many unsupported asser- 
tions, offered without citations. The sources 
that are cited tend to be those published by 
school choice advocates and generally not 
subjected to peer review. Moreover, these 
discussions fail to introduce and discuss 
prominent contradictory research literature. 
The following are some examples: 

• The Heartland report states that gradua- 
tion rates in choice schools in Milwau- 
kee’s voucher program were higher than 
in non-choice schools. But the study 
making this claim was published without 
peer review by School Choice Wiscon- 
sin, a voucher advocacy organization, 
and it was criticized by an earlier think 
tank review as a comparison of “apples 
and oranges.” 3 That is, it compares stu- 
dents using vouchers to their non- 
choosing public school counterparts, 
without accounting for the differences 
due to selection bias and differences be- 



tween the schools, such as graduation 
requirements and student demographics. 4 

• The report also argues that there was 
high parental satisfaction for school 
choice in Florida’s McKay Scholarship 
(voucher) Program. Setting aside the 
question of whether the popularity of 
voucher programs should also be meas- 
ured by asking nonparticipants, the cited 
report has other problems. As was the 
case with the Milwaukee study, it also 
did not go through a peer-review proc- 
ess. Further, it used satisfaction levels as 
reported on a survey to measure the suc- 
cess of the program but did not discuss 
the methodological problems (e.g., se- 
lection bias) of doing so. 5 

• Perhaps most importantly, the report 
asserts that charter school students have 
been shown to be more likely than tradi- 
tional public school students to be profi- 
cient in reading and math. Cherry-picked 
studies can indeed show positive (or 
negative) effects on test scores. But Mi- 
ron, Evergreen, and Urschel (2008) re- 
cently analyzed evidence of the relation- 
ship between charter schools and student 
achievement and found that “on the 
whole, charters perform similarly to tra- 
ditional public schools.” 6 Their compre- 
hensive review of charter school studies, 
which included the most recent high- 
quality studies, yielded mixed results on 
charter school performance, in contra- 
diction to the overly rosy picture painted 
by the new Heartland report. 

V. Review of the Report’s 
Methodologies 

The report does an admirable job explaining 
the criteria used in the grading system. 
States with more school choice options and 
fewer restrictions in choice programs re- 
ceived higher scores. Grades are based on 23 



http://epicpolicy.org/thinktank/review-choice-education 



Page 2 of 5 



dichotomous questions, with a “yes” answer 
worth one point and a “no” answer worth 
zero points. These questions focus on the 
school choice initiatives of states and are 
categorized into four different types of 
school choice (vouchers, charters, tax cred- 
its, and public school choice). 

One concern with this method is that public 
school choice is the subject of fewer ques- 
tions 7 ; a state’s participation in this type of 
choice will generate fewer points (and, rela- 
tively speaking, a lower grade). If a state 
implements vouchers, it can receive up to 
seven points, while participation in public 
school choice will only award it a maximum 
of three points. This aspect of the scoring 
system was not as apparent or well- 
explained as was the overall focus. 

VI. Review of the Validity of the 
Findings And Conclusions 

Since the report’s findings are based solely 
on questions asking whether states are im- 
plementing different types of school choice 
or restricting school choice in certain ways, 
and since there are only 23 questions, the 
report is not comprehensive. These simple 
yes-no questions, all worth the same value, 
make no attempt to determine how states are 
employing the different types of school 
choice, nor do they attempt to assess the 
quality or results of those choice programs. 
Furthermore, as the report acknowledges, 
“each state has different needs, socio- 
economic considerations, constitutional re- 
quirements, and political circumstances,” 
which often call for different school choice 
programs (p. 5). 

Accordingly, a state with little demand for 
school choice would be awarded a better 
grade by the Heartland standard if it adopted 
poorly functioning, underfunded, ill- 
received choice programs that resulted in 



lower student performance. While the report 
deserves praise for the straightforward na- 
ture of grading system, those simple criteria 
start to look questionable if the reader fails 
to accept the premise that a more unre- 
stricted marketplace of choice will necessar- 
ily have a positive outcome. 

The report acknowledges some of these 
limitations, but readers can still be misled 
when making sense of these grades, as dis- 
cussed below. 

VII. Usefulness of the Report 
for Guidance of Policy 
and Practice 

The Heartland Institute advertised its report 
on its website by featuring several news 
releases that describe the report card as one 
that “ranks state efforts to increase account- 
ability and improve student achievement 
with four kinds of school choice: vouchers, 
charters, tax credits, and public school 
choice.” 8 These news releases are tailored to 
individual states based on the grade re- 
ceived. For instance, the headline created for 
media outlets in South Dakota is “Choice 
and Education: South Dakota Earns an F,” 
and the article begins by asserting that “Par- 
ents hoping to wrest control of their chil- 
dren’s futures away from education bureau- 
crats and politicians should consider moving 
from South Dakota.” 9 

Such news releases help to explain why 
Choice and Education across the States is 
limited in its usefulness. Report cards can be 
arbitrary and misleading. According to a 
peer-reviewed study published earlier this 
year by Chi and Weiner, these grades and 
rankings “have great potential to cheapen 
the discourse around important issues.” 10 A 
given ranking system’s particular choice of 
criteria will determine the results. 11 Because 
of these inconsistencies, any given report 
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card is only useful if readers understand the 
ranking criteria and underlying beliefs of its 
authors. But most readers see only news 
releases and headlines, never understanding 
the underlying criteria. 

The article by Chi and Weiner argues that 
grades or rankings should “clearly state limi- 
tations and explain underlying values and 
assumptions.” 1 ' As mentioned above, the 
new Heartland report card does some of this; 
it provides straightforward scoring proce- 
dures and results, and it explains that states 
with more school choice and fewer restric- 
tions received higher scores. But the Heart- 
land news releases do not explicitly explain 
the ranking system and its criteria. They do 
not, for instance, mention that those without 
restrictions are given higher grades. 

According to Chi and Weiner, “[b]y issuing 



report cards, rankings, and grades, organiza- 
tions have found that they can make their 
primary arguments easily understood and 

1 o 

can gain influence in policy debates.” Re- 
port cards also “allow a newspaper an obvi- 
ous headline and an easy local angle.” 14 
Grades provide conclusions that are easy to 
understand and explain, especially for policy 
makers who want quick assessments. In this 
case, the Heartland Institute hopes that this 
report card will “provide a roadmap for state 
legislators seeking to improve student 
achievement through school choice. ” 1:> But it 
is unclear what these state legislators could 
learn from the new Heartland report, other 
than that their state was just given a (low) 
grade. If their new understanding is merely 
that the school choice system in their state is 
less extensive than that in another state, then 
it would seem the report is of use only as an 
advocacy document. 
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